What is the fastest way to extract data from a huge text file?

Prashant Kumar answered . 2024-12-21 02:42:01

• less than a tenth of a second to read and parse the sample file (with the file in the system cache)
• less than a tenth of a millisecond to retrieve one value
• the array ION is half a MB. Make ION uint8 to save memory - if needed.
• 62196 values retrieved from the sample file.

    >> tic,ION = cssm();toc
    Elapsed time is 0.074765 seconds.
    >> sum(not(isnan(ION(:))))
    ans =
           62196
    >> whos ION
      Name       Size                Bytes  Class     Attributes

      ION       73x71x12            497568  double            

    >> ION(lon2ix(0),lat2ix(85),ut2ix(20))
    ans =
       164  

    >> tic,ION(lon2ix(0),lat2ix(85),ut2ix(20));toc
    Elapsed time is 0.000067 seconds.

compared to

    >> tic, [gim_tec] = sample_search_function( 20, 85, 0 ), toc
    gim_tec =
       164
    Elapsed time is 0.265756 seconds.

where

    function    ION = cssm()
    str = fileread( 'c:\m\cssm\CODG1520.txt' );
    ca1 = regexp( str, '(?<=START OF TEC MAP).+?(?=END OF TEC MAP)', 'match' );
    ION = nan( 73, 70, 11 );
    lat2ix  = @(lat) round((lat+87.5)/2.5)+1;
    lon2ix  = @(lon) round((lon+180)/5.0)+1; %#ok
    ut2ix   = @(ut)  round(ut/2)+1;
    for jj = 1 : length( ca1 )

        buf = regexp( ca1{jj}, '\n', 'split', 'once' );
        buf = regexp( buf{2} , '\n', 'split', 'once' );
        ut  = textscan( buf{1}, '%*f%*f%*f%f%*[^\n]' );
        ut  = ut{1};
        ca2 = regexp( buf{2}, 'LAT/LON1/LON2/DLON/H', 'split' );
        pos = ca2{1};
        for kk = 2 : length( ca2 )
            lat = textscan( pos,'%f%*[^\n]' ); 
            lat = lat{1};
            num = sscanf( ca2{kk}(1:end-60), '%f' );
            pos =  strtrim( ca2{kk}(end-60+1:end) );
            ION(:,lat2ix(lat),ut2ix(ut)) = num;
        end
    end
    end

Not satisfied with the answer ?? ASK NOW