Add a comment

 

Re: The joy of sets

I've seen this kind of problem more times than I can count. I once worked on a batch process where the code exported data, pivoted it and then reimported it back to the DB clearly because the original coder didn't understand SQL/set processing properly. I changed the code to use SQL and reduced the batch running time from 12 hours to 6 minutes. The standard ETL process (whether using a fancy tool or not) works very well. 1. Get the data into a table with no constraints/checks 2. Normalise and process the data, look up foreign keys, perform validations etc into loading tables (not the production tables) 3. Once all the validation has been done move the data into the main production tables I'd highly recomend always following a process like this for importing 'untrusted data'

Re: The joy of sets


Title
Body
HTML : b, strong, i, em, blockquote, br, p, pre, a href="", ul, ol, li, sub, sup
Name
E-mail address
Website
Remember me Yes  No 

E-mail addresses are not publicly displayed, so please only leave your e-mail address if you would like to be notified when new comments are added to this blog entry (you can opt-out later).