I guess everyone has that chore they hate doing. Mine is laundry. I approach it with dread. I feel the same way about preparing data for analysis. It drives me crazy that with 30+ years of computer aided manufacturing experience under our belts we still struggle mightily with merging data from multiple systems and cleaning the data prior to analysis.
The problem is getting worse as business requirements demand faster and better decisions. The process industry’s big data problem is the need to support the interlocked decisions that drive profitability. We need to coordinate the hundreds of decisions made regarding the supply chain, the product assets and the myriad other facets of managing the business.
Managing the data supporting those decisions is a labor intensive process. Data are extracted from individual systems and then merged. Then, false values and outliers are removed and any required transformations are completed. Once those tasks are complete, the analysis is performed. The findings are discussed and an action plan is formed. The timeframe for this workflow is hours if not days.
Is there finally a solution to simplifying these tasks? Check out this short video.