Databricks Unleashes AI SQL Functions with Foundation Models

Databricks is shaking up the world of data analytics with its innovative AI SQL functions, which are powered by foundation models. These tools seamlessly blend into your existing data workflows, making it easier than ever to conduct AI-powered analysis directly within your SQL queries. This means you can say goodbye to the hassle of complex API integrations and hello to a more streamlined approach to data analysis. This article explores how these new features could change the way you handle data.

Key Points

• Databricks introduces AI SQL functions powered by foundation models.
• These functions offer capabilities such as sentiment analysis, classification, text extraction, and grammar correction.
• AI models are integrated into Databricks, removing the need for external API calls.
• The functions are available on a pay-per-token basis, providing cost-effective AI solutions.
• Availability is restricted to specific Databricks regions.
• The AI extract function enables versatile information extraction.
• The AI fix grammar function enhances the grammatical quality of text.

Databricks AI SQL Functions: Foundation Models in Action

Understanding the New Databricks AI SQL Functions

Databricks has rolled out a suite of AI SQL functions that bring AI directly into SQL queries, simplifying the process for data analysts and engineers. These functions tap into the power of foundation models, which are large language models trained on vast datasets. Typically, such models, like ChatGPT, are hosted externally and require complex API integrations. Databricks changes this by embedding these models right into their platform, making AI more accessible and efficient. Instead of each company building its own web service calls, Databricks offers off-the-shelf models, saving time and resources.

While these functions make AI integration easier, it's important to consider the costs. They operate on a pay-per-token basis, so strategic implementation is key to manage expenses effectively. Also, these functions are not available in all Databricks regions yet, currently limited to regions like Central US, East US, East US 2, and North Central US. This might affect your workspace and project planning.

Core AI SQL Functions: Sentiment Analysis, Classification, and More

Databricks offers several core AI SQL functions that enrich data analysis:

• AI Analyze Sentiment: Determine the sentiment (positive, negative, or neutral) in text, which is great for understanding customer feedback and social media trends.
• AI Classify: Categorize text into predefined classes, such as sorting customer inquiries by topic or product.
• AI Extract: Pull specific information, like names or email addresses, from unstructured text. This turns raw text into structured data, ideal for creating detailed models.



• AI Fix Grammar: Correct grammar in text, useful for cleaning up user-generated content or ensuring professional communication.
• AI Mask: Protect privacy by masking sensitive information in text, aiding compliance with data security regulations.
• AI Summarize: Create concise summaries of long documents or articles, perfect for extracting key information quickly.
• AI Translate: Translate text between languages, expanding your data analysis capabilities across different sources.
• AI Similarity: Calculate similarity scores between records, allowing for more refined data analysis.

Navigating Databricks AI SQL Functions Using Databricks Foundation Model APIs

The following table lists the Databricks AI SQL functions powered by Databricks Foundation Model APIs:

Practical Applications: Real-World Use Cases

These AI SQL functions open up a range of practical applications:

• Analyze Customer Reviews: Automatically assess the sentiment in customer reviews to pinpoint areas for improvement and gauge customer satisfaction.



• Automate Data Quality Checks: Use AI grammar correction to clean up data before analysis, including AI summarization for efficiency.
• Streamline Document Processing: Extract key information from legal documents, contracts, or research papers, populating other datasets with extracted data like names and addresses.

By allowing AI functions to run directly in SQL, data workflows become significantly more streamlined.

Putting AI SQL Functions to the Test: A Demo in Databricks

Databricks offers a compelling demonstration of these functions in action:

• Sentiment Analysis: With a simple SQL query, you can analyze text sentiment:
SELECT ai_analyze_sentiment('I am a happy sparky boi');
• Data Cleaning: Query different data points to analyze customer reviews and perform various checks, making data cleaning more effective.
• Using a common table expression (CTE) to store data, you can analyze different users' sentiments and perform various analyses.
• Switching to AI fix grammar can help correct common grammatical issues.

Availability and Prerequisites for Using Foundation Models

To use Foundation Models effectively, certain requirements must be met:

• AI functions are only available on workspaces in Foundation Model APIs pay-per-token supported regions.
• This function is not available on Azure Databricks SQL Classic.

Tips for Maximizing the Potential of Databricks AI SQL Functions

Strategic Integration

Plan carefully how to integrate these AI functions into your data pipelines. Identify where AI insights can add the most value and automate repetitive tasks. Use these functions where they will provide the most benefit.

Cost Optimization

Keep an eye on your token consumption to optimize costs. Assess current spending and explore other AI options to enhance functionality while evaluating the trade-off between AI-driven insights and costs.

Stay Updated

Databricks is continually improving its AI capabilities. Stay informed about new functions, region availability, and pricing changes to fully leverage this transformative technology. Keep learning and experimenting to enhance your workflows.

Advantages and Disadvantages of Databricks AI SQL Functions

Pros

• Simplified AI integration within SQL workflows
• Access to powerful foundation models
• Reduced complexity compared to external API integrations
• Potential for cost-effective AI solutions (pay-per-token)
• Automated Data Quality Checks

Cons

• Cost management (pay-per-token usage)
• Limited region availability
• Dependence on Databricks' platform
• Potential vendor lock-in
• Specific workspace and configuration requirements

Frequently Asked Questions

What are Databricks AI SQL functions?

Databricks AI SQL functions are a suite of tools that allow you to leverage AI models directly within your SQL queries. These functions are powered by foundation models and offer capabilities like sentiment analysis, text classification, and more.

What types of functions are AI extract and AI fix grammar?

The AI extract function enables term extraction and document parsing to pull out specific information like emails or names. The AI fix grammar function corrects grammatical errors in text.

How does pricing work for these AI functions?

Pricing is based on a pay-per-token model. Token usage depends on the complexity of the query and the size of the input text. For detailed pricing information, refer to the Databricks documentation.

In which regions are Databricks AI SQL functions available?

Currently, these functions are available in specific regions including Central US, East US, East US 2, and North Central US. Always check the latest Databricks documentation for the most current region support.

Related Questions

What are the alternatives to performing sentiment analysis in Databricks?

Before Databricks SQL functions, businesses had a few options. One was to build a model from scratch using their data to create a custom classification engine. Another was to use web service calls, which required setting up a separate subscription, potentially exposing the organization to data leaks or other concerns.