Python: Optimizing a generator
up vote
1
down vote
favorite
For large v_max and v_max_n, the generator below yields very slow when I have 750000000 and 500000000, respectively. I have been trying different ways for days and cannot seem to figure this out.
from itertools import product
n = 15 * 10 ** 8
v_max = math.floor(n / 2 - 1 / (2 * n))
v_max_n = n // 3
(i for i in product(range(2, v_max_n + 1, 2),
range(2, v_max + 1, 2),
range(3, v_max + 1))
if i[0] <= i[1] and i[1] <= i[2] and sum(i) <= n and
i[0] ** 2 + i[1] ** 2 == i[2] ** 2 - 1)
I have tried using it as map(lambda x: x, gen), nested for loops, and have generally been racking my brain to no avail.
python performance iterator generator
asked 24 mins ago
dustin
366
|
show 1 more comment
up vote
1
down vote
favorite
For large v_max and v_max_n, the generator below yields very slow when I have 750000000 and 500000000, respectively. I have been trying different ways for days and cannot seem to figure this out.
from itertools import product
n = 15 * 10 ** 8
v_max = math.floor(n / 2 - 1 / (2 * n))
v_max_n = n // 3
(i for i in product(range(2, v_max_n + 1, 2),
range(2, v_max + 1, 2),
range(3, v_max + 1))
if i[0] <= i[1] and i[1] <= i[2] and sum(i) <= n and
i[0] ** 2 + i[1] ** 2 == i[2] ** 2 - 1)
I have tried using it as map(lambda x: x, gen), nested for loops, and have generally been racking my brain to no avail.
python performance iterator generator
asked 24 mins ago
dustin
366
For me this does not even fit into memory (16 GB). Which is weird, sinceproductshould not make any intermediate lists and neither shouldrange...
– Graipher
15 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
1
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago
|
show 1 more comment
up vote
1
down vote
favorite
up vote
1
down vote
favorite
For large v_max and v_max_n, the generator below yields very slow when I have 750000000 and 500000000, respectively. I have been trying different ways for days and cannot seem to figure this out.
from itertools import product
n = 15 * 10 ** 8
v_max = math.floor(n / 2 - 1 / (2 * n))
v_max_n = n // 3
(i for i in product(range(2, v_max_n + 1, 2),
range(2, v_max + 1, 2),
range(3, v_max + 1))
if i[0] <= i[1] and i[1] <= i[2] and sum(i) <= n and
i[0] ** 2 + i[1] ** 2 == i[2] ** 2 - 1)
I have tried using it as map(lambda x: x, gen), nested for loops, and have generally been racking my brain to no avail.
python performance iterator generator
asked 24 mins ago
dustin
366
For large v_max and v_max_n, the generator below yields very slow when I have 750000000 and 500000000, respectively. I have been trying different ways for days and cannot seem to figure this out.
from itertools import product
n = 15 * 10 ** 8
v_max = math.floor(n / 2 - 1 / (2 * n))
v_max_n = n // 3
(i for i in product(range(2, v_max_n + 1, 2),
range(2, v_max + 1, 2),
range(3, v_max + 1))
if i[0] <= i[1] and i[1] <= i[2] and sum(i) <= n and
i[0] ** 2 + i[1] ** 2 == i[2] ** 2 - 1)
I have tried using it as map(lambda x: x, gen), nested for loops, and have generally been racking my brain to no avail.
python performance iterator generator
python performance iterator generator
asked 24 mins ago
dustin
366
asked 24 mins ago
dustin
366
edited 7 mins ago
asked 24 mins ago
dustin
366
asked 24 mins ago
dustin
366
asked 24 mins ago
dustin
366
366
For me this does not even fit into memory (16 GB). Which is weird, sinceproductshould not make any intermediate lists and neither shouldrange...
– Graipher
15 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
1
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago
|
show 1 more comment
For me this does not even fit into memory (16 GB). Which is weird, sinceproductshould not make any intermediate lists and neither shouldrange...
– Graipher
15 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
1
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago
For me this does not even fit into memory (16 GB). Which is weird, since
product should not make any intermediate lists and neither should range...– Graipher
15 mins ago
For me this does not even fit into memory (16 GB). Which is weird, since
product should not make any intermediate lists and neither should range...– Graipher
15 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
1
1
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago
|
show 1 more comment
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
});
});
}, "mathjax-editing");
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "196"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f209821%2fpython-optimizing-a-generator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Code Review Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f209821%2fpython-optimizing-a-generator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
For me this does not even fit into memory (16 GB). Which is weird, since
productshould not make any intermediate lists and neither shouldrange...– Graipher
15 mins ago
@Graipher you should just get a generator object when you initialize the generator which shouldn't use any memory
– dustin
12 mins ago
I agree. Yet it does consume memory on my machine (even with Python 3.6.3).
– Graipher
11 mins ago
@Graipher I am on 3.6.6 Anaconda on Mac. Only 8 GBs of RAM, I checked my memory when creating the object but didn't have that problem.
– dustin
9 mins ago
1
@Graipher sum(i) <= n. Added the variable definitions.
– dustin
7 mins ago