Dennes Torres

Featured in:

Articles

Fabric Copy Job: What's new in it

Jan 15, 2025 | red-gate.com | Dennes Torres

At first sight, the new Copy Job may seem only one more redundancy: Why do we need this? We have Data Pipelines for the Copy Activity and also a Copy Assistant to help us configure it. What’s new about the Copy Job? Spoiler: It’s much more than a new UIThe new Copy Job Object and UIThe Copy Job is exactly what the name says: a “dedicated data pipeline” intended only to execute a copy activity. We schedule it exactly as a data pipeline.
High Concurrency Data Pipelines in Fabric

Jan 8, 2025 | red-gate.com | Dennes Torres

Data Pipelines can orchestrate many activities, creating a flow for data ingestion. One of these activities is the notebook execution activity. However, every time a data pipelines executes a notebook, it creates a completely new session and spark pool. This makes the Data Pipeline very slow and expensive. How bad it can beImagine your pipeline will run a notebook inside a loop. The loop executes the notebook many times. Each execution means a completely new spark pool. This is expensive.
T-SQL Tuesday #181: Query Store and its evolution

Dec 9, 2024 | red-gate.com | Dennes Torres

Query Store. This is my choice in relation to “What feature release made you say WOW!”, the subject of this SQL Tuesday. Query Store: The first releaseQuery Store was first released in SQL Server 2016, it’s getting closer to 10 years old, but we barely notice. The starting idea was something simple, a common requirement and yet capable of making everyone say “WOW”. Query tunning before query store was based on the current query activity, or the query plan cache.
Granting Permissions in a Fabric Data Warehouse and Lakehouse

Dec 4, 2024 | red-gate.com | Dennes Torres

We are becoming used to being a bit lazy when granting permissions to Data Warehouses and lakehouses in Fabric. We only go to the workspace level and add the user as a viewer or member. However, this is far from a good idea. This practice not only gives the user access to all tables in the warehouse or lakehouse, but also gives access to all objects in the workspace. Bad idea. The SolutionThe solution is on the old and good SQL security practices.
Breaking Down Ignite: A Pragmatic Look at the Conference’s Best Innovations

Nov 21, 2024 | red-gate.com | Dennes Torres

The amount of news brought by Ignite is huge. I was not expecting to find so many new features, resources, and discoveries. There may be dozens of summaries of the conference, and each one focusing on different highlights and news. On this article I will summarize what seems to be the most groundbreaking changes but bringing a pragmatic view of each one of them. The new announcements are not simple features.
KQL: The top of a group in an easy way

Nov 12, 2024 | red-gate.com | Dennes Torres

When learning something new, we always compare it with what we know. In this case, we end up comparing KQL with SQL. In SQL, when we need to get the top rows based on a grouping, the process is not easy. For example, let’s say that in a table containing taxi rides, we want to retrieve the record of the ride with the highest fare on each day. There are multiple ways to do this, none is too easy. One of the methods is to create a row_number based on the day.
Analyzing Kusto Query History in Fabric

Nov 7, 2024 | red-gate.com | Dennes Torres

We can consume a Kusto database in Fabric from many different places: Notebooks, semantic models, real time dashboards and more. Kusto register all queries sent by the consumers in the query history. Sometimes, either for logging purpose or to analyze and fix some bug, we need to identify the queries the database is receiving and executing. Checking the Query History in Kusto DatabaseThe statement is simple: However, a simple execution may generate an error.
T-SQL Notebooks: Manage Views Source Control and more

Oct 30, 2024 | red-gate.com | Dennes Torres

T-SQL Notebooks is one of the new features announced during FabCon Europe. The most distracted could miss the fact this is a new feature at all. Yes, it is. Notebooks were capable to support Spark SQL, but T-SQL is something new. The main examples being announced are built with data warehouses, but let me confirm and highlight this:T-SQL Notebooks support lakehouses as well. There is at least one limitation: DML is not supported with lakehouses. Let’s start from the beginning.
Shortcuts, Capacities and Costs : Some Considerations

Oct 23, 2024 | red-gate.com | Dennes Torres

Let’s start with a question:Question: Where is the cost charted when one or more shortcuts in a lakehouse points to a table located in a different capacity ? From the capacity which holds the shortcut or from the capacity which holds the data? Answer: The charges go to the capacity holding the data, the target of the shortcutThis simple feature brings many interesting consequences and possibilities.
Using a Service Account in Microsoft Fabric

Oct 16, 2024 | red-gate.com | Dennes Torres

Applications should never depend on user accounts for their own authentication. User accounts have unique security configurations, such as MFA and password expiration. Applications can’t manage this kind of unique configuration. When the user changes the password, for example, the application may stop. The solution for this is to give to the application their own identity. The application needs to be capable to use this kind of identity, which is called a Service Account.