Posts

Showing posts with the label dedup component

S) scan ,rollup and dedup with null key and unique key

Image
  Scan with null key ---11 rec   Rollup with null key –1 record( depends if the data is sorted then last if data is not sorted then fst record)   Dedup with null key —( keep fst )—1 record   Dedup with null key —( keep lst )—1 record   Dedup with null key —( keep unique)—0 record ( key didn’t get the unique record bcoz it treats every record is one group      

S) Dedup Example

Image
  input : -   id   1   1   1   2   2   3     Output required   id   1   1   2     Id   3   Id   1   2   Note : - Use dedup keep fst and keep last with null key .            

S) How to remove header and trailer from CSV file?

Read by new line initially and remove header trailer then redefine. To remove trailer, use dedup component with option key as null and keep last record. Finally retrieve all data records from dedup port