Krdytkyu

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use stochastic gradient descent I'm getting really poor accuracy so I'm not sure if it's my learning rate, epochs or just my code is incorrect. Also I'm wondering how would I add regularization to both of these? Do I add a variable lambda and multiply it by the learning rate or is the more to it?

BGD:

def batch_gradient(df, weights, bias, lr, epochs):

    X = df.values

    y = X[:,:1]

    X = X[:,1:]

    length = X.shape[0]

    for i in range(epochs):

        output = (sigmoid((np.dot(weights, X.T)+bias)))

        weights_tmp = (1/length) * (np.dot(X.T, (output - y.T).T))

        bias_tmp = (1/length) * (np.sum(output - y.T)) 



        weights -= (lr * (weights_tmp.T))

        bias -= (lr * bias_tmp)



    return weights, bias

SGD:

def stochastic_gradient(df, weights, bias, lr, epochs):

    x_matrix = df.values

    for i in range(epochs):

        np.random.shuffle(x_matrix)

        x_instance = x_matrix[np.random.choice(x_matrix.shape[0], 1, replace=True)]

        y = x_instance[:,:1]



        output = sigmoid(np.dot(weights, x_instance[:,1:].T) + bias)    

        weights_tmp = lr * np.dot(x_instance[:,1:].T, ((output - y)))



        weights = (weights - weights_tmp.T)

        bias -= lr * (output - y)



    return weights, bias