Krdytkyu

I'd like to calculate some statistics of given data (frequencies of difference among elements on various distances in percent multiplied by 10) using moving window within that data. Is it possible to speed up the code below? I noticed that some calculations are repeating. But I was not able to exclude them without additional slowness.

def get_dist_stat(pdata, pwin_length):

    ''' pdata - given data array

        pwin_length - the lenght of window

        the function returns stat table where

            row represents the distance between elements

            col represents the difference for that distance in percent multiplied by 10 (assume that maximum difference can be 20 percent)



    '''



    l_data = len(pdata)

    l_win = pwin_length

    print("l_data=", l_data)

    print("l_win=", l_win)



    # stat table

    stat_table = np.zeros((l_win-1, 20*10), dtype = int)



        # loop over all data

    for k in range(l_data - l_win + 1):



        win = pdata[k : k + l_win]

        print('-' * 10)

        print("k=", k, " kend=", k + l_win )



        print("win=", win)



        # loop over window

        for i  in range(1 , l_win):

            b=win[i:]

            a=win[:-i]

            diff=(abs((b-a)/a*100 ) * 10).astype(int)

            print("i=",i)

            print("b=", b)

            print("a=", a)

            print("diff=",diff)



            # storing found differences into stat table

            apercents, acount = np.unique(diff, return_counts = True)

            l_apercents = len(apercents)

            for j in range(l_apercents):

                stat_table[i-1, apercents[j]] += acount[j]

    return stat_table



adata=np.array([1.1,1.2,1.3,1.4,1.5])

print("adata=", adata)



astat_table=get_dist_stat(adata,3)

print(astat_table)

And that is its output

adata= [1.1 1.2 1.3 1.4 1.5]

l_data= 5

l_win= 3

----------

k= 0  kend= 3

win= [1.1 1.2 1.3]

i= 1

b= [1.2 1.3]

a= [1.1 1.2]

diff= [90 83]

i= 2

b= [1.3]

a= [1.1]

diff= [181]

----------

k= 1  kend= 4

win= [1.2 1.3 1.4]

i= 1

b= [1.3 1.4]

a= [1.2 1.3]

diff= [83 76]

i= 2

b= [1.4]

a= [1.2]

diff= [166]

----------

k= 2  kend= 5

win= [1.3 1.4 1.5]

i= 1

b= [1.4 1.5]

a= [1.3 1.4]

diff= [76 71]

i= 2

b= [1.5]

a= [1.3]

diff= [153]

[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1

  0 0 0 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0

  0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

You've already made the key observation here, which is that most of the work is redone. Each time you pick a window, most of the calculations are the same as the previous window.

In fact it's much faster to do all the calculations ahead of time into one big ndarray, and then for each window, pick out the calculations that are relevant. So we don't need the temporary a and b lists.

How many dimensions do we need? Just starting point and length. It's going to be a triangular array, so we'll waste some space.

precomputed_results = np.zeros(l_win+1, l_data), dtype = int)

# First pass

for interval in range(1, l_win):

    for first_point_index in range(l_data-interval): 

        # compute diff relative to elements [first_point_index] and [first_point_index+interval]

        # line will be similar to precomputed_results[...] = ...



# Second pass

for interval in range(1, l_win):

    for first_point_index in range(l_data-interval):

        # use slicing on precomputed_results