Handling Large Data in Salesforce

 Managing large data volumes in Salesforce necessitates employing effective strategies and optimization techniques. Below are some recommended approaches to handle this situation efficiently:

1. Data Archiving and Purging: Identify historical or redundant data and archive or delete it from the Salesforce instance. This practice helps reduce the overall size of your database and enhances performance.

2. Data Model Optimization: Design your Salesforce data model in a manner that minimizes the number of records and maximizes data reuse. Use appropriate relationships, such as lookup or master-detail relationships, to avoid data duplication.

3. Indexing: Index critical fields used in search filters or report criteria. Indexing can significantly enhance query and search performance on large data sets.

4. Bulk API: Utilize the Salesforce Bulk API for operations involving large data volumes, such as data imports, updates, or deletions. The Bulk API enables processing large data sets in batches, leading to improved performance.

5. Asynchronous Processing: Consider leveraging Salesforce Asynchronous Apex or Queueable Apex for resource-intensive operations. These mechanisms enable the management of long-running processes without adversely affecting the user experience.

6. Data Loader and External Tools: Take advantage of tools like the Salesforce Data Loader or third-party ETL (Extract, Transform, Load) tools to handle large data volumes efficiently. These tools facilitate data migration, synchronization, and manipulation tasks at scale.

7. Governor Limits and Limits Management: Ensure that your processes operate within the Salesforce governor limits. Monitor system usage and implement measures to gracefully handle limit exceptions.

8. Data Partitioning: If feasible, partition your data into smaller logical units to enhance manageability. For instance, consider partitioning data based on geographical regions or business units.

9. Data Archiving Services: Salesforce provides data archiving solutions like "Big Objects" that can handle large historical data volumes that are infrequently accessed. Utilize these features to offload older data from your primary tables.

10. Performance Tuning: Regularly monitor your Salesforce instance's performance and optimize slow-performing queries or processes. This may involve creating custom indexes, optimizing code, or reevaluating data access patterns.

Salesforce Connect

 What is Salesforce Connect? 

Salesforce Connect provides seamless integration of data across system boundaries by letting your users view, search, and modify data that’s stored outside your Salesforce organization, without actually copying the data into Salesforce. Salesforce Connect is a point-and-click solution to your other databases, and it makes earlier integration scenarios seem archaic.

Data from external systems are available as External Objects in Salesforce that are similar to Standard and Custom Objects so things like Tabs, Page Layouts, Custom Fields exist.

Salesforce Connect is used most Commonly

1.Searching External Data

2.List Views

3.Record Views

4.Related Lists

Note :

We can’t create formula fields, validation rules, and workflows on External Objects.

Salesforce Connect Cross Org :

Salesforce Connect Cross Org is one of the adapter types available in Salesforce Connect. When you define an external data source in your organization, you specify the Salesforce Connect Cross Org adapter in the Type field, and this will allow you to connect your org to another org, and access the records of this other org in your current Salesforce org.

(OR)

With the extension of Salesforce Connect Cross Org Adapter in the external data source, the users can create, edit, delete the records in/from the external system with support of writable interface.

Get Salesforce System users using SOQL - Tip

 Salesforce doesn't let you access the Automated Process and System User from the Salesforce UI. However, you can use SOQL to get the information on these two Users.

SELECT Id,ProfileId,Name,Profile.Name,Alias,Email,Createddate,IsActive FROM User WHERE Name =’System’ OR Name=‘Automated Process’

Salesforce non-deterministic formula fields

 Here are examples of common non-deterministic formulas. Force.com cannot index fields that:

• Reference other entities (i.e., fields accessible through lookup fields)
• Include other formula fields that span over other entities
• Use dynamic date and time functions (e.g., TODAY, NOW)

A formula is also considered non-deterministic when it includes:

• Owner, autonumber, divisions, or audit fields (except for CreatedDate and CreatedByID fields)
• References to fields that Force.com cannot index
• Multi-select picklists
• Currency fields in a multicurrency organization
• Long text area fields
• Binary fields (blob, file, or encrypted text)
• Standard fields with special functionalities
• Opportunity: Amount, TotalOpportunityQuantity, ExpectedRevenue, IsClosed, IsWon
• Case: ClosedDate, IsClosed
• Product: Product Family, IsActive, IsArchived
• Solution: Status
• Lead: Status
• Activity: Subject, TaskStatus, TaskPriority

Master Data Management (MDM)

 There are four different MDM implementation styles 

—> The Registry style

—> The Consolidation style

—> The co-existence style

—> The Centralised style

Registry :

—> Data is authored in different systems

—> Data is then sent to the MDM

—> MDM deduplicates data

—> Data cleansed using match and merge.

—> Data is not sent back to source systems.

—> All systems data can now be analysed and duplicates can be spotted in a central place.

Consolidation :

—> Data is authored in different systems

—> Data is then sent to the MDM

—> MDM deduplicates data

—> Data cleansed using match and merge

—> Data is not sent back to source systems

—> Used as a high quality DW/BI system for reporting and analytical purposes.

Coexistence :

—> Data is authored in different systems

—> Data is then sent to the MDM

—> MDM deduplicates data

—> Data cleansed using match and merge

—> Clean Data is sent back to source systems

—>Single version of truth between MDM and source systems.

Centralized :

—> Master data is authored in MDM,not in the systems.

—> All systems subscribe to the MDM hub for Master data.

—> MDM is the sole source of data.

—> Systems do not write back data to MDM.

	Registry	Consolidation	Coexistance	Centralized
Source of data	Systems	Systems	Systems	MDM
Complexity	Low	Medium	High	High
Single Source of Truth	MDM	MDM	MDM + Systems	MDM
Data updated back?	No	No	Yes	MDM
Best used for	Data analysis for compliance	BI/Reporting	Distributed Source of Truth	Top down approach

System of Record (SOR) :

—> The authoritative data source for a given data element or piece of information.

—> The data repository where the data object, as a whole or specific attributes of a data object, are maintained.

—> This includes data creation, updating, modifying and deleting.

System of Record Characteristics 

—> Durable

       Eliminate single point of failure

—> Correct

       Storage technology

—> Restorable

      Be able to restore backed-up data

—> Disaster-Ready

       Be able to switch to a DR site in case of a disater

Golden Record :

In some cases, there is no single system that has the complete data set.

—> Bits and pieces of the required data is stored in multiple systems.

—> We cannot easily combine the bits and pieces to make a complete meaningful data set.

In other cases, the same data with the same attributes are available in multiple systems

—> But, the data values are different and we are not sure which among them is accurate.

The solution would be to create a ‘Golden Record’ in a single source of truth (SSOT) by compiling element attributes from the different systems.

MDM creates a master record ( also known as a ‘Golden Record’ or ‘Best Version of the Truth’ )that contains the essential information upon which a business or organization relies.

The ‘Golden Record’ contains what an organization needs to know about critical ‘things’ - a customer , location, product, supplier and so on.

Single Source of Truth (SSOT) :

—> The Single source of Truth (SSOT) or sometimes called Golden Source of Truth is a trusted data source that hosts a complete picture of a given data element.

—> It contains the ‘Golden Records’.

—> It can be used as source for any Business Intelligence and Data warehouse system.

—> AKA Golden source of Truth.

Deduplication Approaches

—> Survival of the fittest record

—> Use of Data Survivorship Rules

Survival of the fittest record :

Selecting the record that according to a data quality rule is the most fit is the simplest approach. The rule(s) that determines which record that will survive is most often based on either :

—> Lineage, where the source systems are prioritised : the record that belong to the highest priority source will be selected as a golden record.

—> Completeness - which record has the most fields and characters filled : the record with the highest percentage of completed fields will be selected as a golden record.

The downside of this approach : the surviving record has values from 1 source but not the other.

Use of Data Survivorship Rules :

In order to determine which field values should be selected while establishing the golden record in the master hub, it will be necessary to define data survivorship rules.

Data Survivorship Rules are set of rules that can be applied on the same data element that originates from different source systems to determine the golden record version of this data element.

Data Survivorship Rules factors

Factor	Description
Accuracy	Records from a specific system might have a higher accuracy than records in any other system regardless of all other factors
Recency	A record that was created more recently, or has a more recent update is more reliable than another record that was updated years ago.
Frequency	A field value that is the same in several systems is more reliable than another value that appears once in another system
Completeness	A record that has more complete field is more reliable than another record that has less complete fields.