Working with large black/whitelists
Working with large black/whitelists
Greetings!
I work with many dozens of thousands of white/blacklist entries. Every time I want to add a new entry, it has to reload the entire page, this takes between 3 and 6 minutes (depending on my bandwidth). Is it possible to make a change so that it adds new entry to white/blacklist but doesn't have to reload the whole lot? For example, to display a message - "click here to load the entire list" or something similar? This will enable me to add good/bad IP addresses much quicker into my lists.
I work with many dozens of thousands of white/blacklist entries. Every time I want to add a new entry, it has to reload the entire page, this takes between 3 and 6 minutes (depending on my bandwidth). Is it possible to make a change so that it adds new entry to white/blacklist but doesn't have to reload the whole lot? For example, to display a message - "click here to load the entire list" or something similar? This will enable me to add good/bad IP addresses much quicker into my lists.
- shawniverson
- Posts: 3650
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Working with large black/whitelists
Sounds like a good feature request I am noting this to engage MailWatch development.
Re: Working with large black/whitelists
Should be an easy change. Let me have a look and tell you what to do tomorrow.
Re: Working with large black/whitelists
Given the transient nature of the internet in general, do you really feel this is a very good idea to pursue?
Re: Working with large black/whitelists
?
I'd find it useful. There are a few interface tweaks that I make to the mailwatch ui to speed up the interface. For example, when I run message filters to select particular messages, do I really need to know how many messages are going to appear before I go to the message list? Nah, no need to hold up the ui just to do a "select * from maillog where whatever" query.
I'd find it useful. There are a few interface tweaks that I make to the mailwatch ui to speed up the interface. For example, when I run message filters to select particular messages, do I really need to know how many messages are going to appear before I go to the message list? Nah, no need to hold up the ui just to do a "select * from maillog where whatever" query.
Re: Working with large black/whitelists
Easy temporary fix for you:
tl;dr; hate reading diffs
0/ edit lists.php
1/ add this code after line 32
2/ change this line:
to this:
3/ change this section from this:
to this:
4/ change this section from this:
to this (yes, I was too lazy to change the indentation of the whole function):
5/ and finally change this:
to this:
To make your list reappear, just add a ?showall to your url, so from this:
to this:
Note that is is just a temporary hack that will disappear on the next efa update. However, it's a trivial fix to add back. Also, if anyone thinks this should be a feature for mailwatch, this is not the right way to do it. It should be something configurable in the settings and properly localized.
Code: Select all
[root@efa mailscanner]# diff -c lists.php.org lists.php
*** lists.php.org Wed Nov 22 01:27:25 2017
--- lists.php Wed Nov 22 01:28:18 2017
***************
*** 30,35 ****
--- 30,37 ----
html_start(__('wblists07'), 0, false, false);
+ if (isset($_GET['showall'])) { $showall=true; }
+
if (isset($_GET['type'])) {
$url_type = deepSanitizeInput($_GET['type'], 'url');
if (!validateInput($url_type, 'urltype')) {
***************
*** 302,309 ****
* @param string $list
* @return array
*/
! function build_table($sql, $list)
{
$sth = dbquery($sql);
$table_html = '';
$entries = $sth->num_rows;
--- 304,312 ----
* @param string $list
* @return array
*/
! function build_table($sql, $list, $showall=false)
{
+ if ($showall) {
$sth = dbquery($sql);
$table_html = '';
$entries = $sth->num_rows;
***************
*** 325,330 ****
--- 328,336 ----
} else {
$table_html = __('noentries07') . "\n";
}
+ } else {
+ $table_html = "Results hidden\n";
+ }
return array('html' => $table_html, 'entry_number' => $entries);
}
***************
*** 412,422 ****
$whitelist = build_table(
'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
! 'w'
);
$blacklist = build_table(
'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
! 'b'
);
echo '</table>
</form>
--- 418,430 ----
$whitelist = build_table(
'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
! 'w',
! $showall
);
$blacklist = build_table(
'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
! 'b',
! $showall
);
echo '</table>
</form>
0/ edit lists.php
1/ add this code after line 32
Code: Select all
if (isset($_GET['showall'])) { $showall=true; }
Code: Select all
function build_table($sql, $list)
Code: Select all
function build_table($sql, $list, $showall=false)
Code: Select all
function build_table($sql, $list, $showall=false)
{
$sth = dbquery($sql);
Code: Select all
function build_table($sql, $list, $showall=false)
{
if ($showall) {
$sth = dbquery($sql);
Code: Select all
} else {
$table_html = __('noentries07') . "\n";
}
return array('html' => $table_html, 'entry_number' => $entries);
Code: Select all
} else {
$table_html = __('noentries07') . "\n";
}
} else {
$table_html = "Results hidden\n";
}
return array('html' => $table_html, 'entry_number' => $entries);
Code: Select all
$whitelist = build_table(
'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
'w'
);
$blacklist = build_table(
'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
'b'
Code: Select all
$whitelist = build_table(
'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
'w',
$showall
);
$blacklist = build_table(
'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
'b',
$showall
Code: Select all
https://<efa-server>/mailscanner/lists.php
Code: Select all
https://<efa-server>/mailscanner/lists.php?showall
Re: Working with large black/whitelists
Wow guys, thanks very much! I will try it out now!
- shawniverson
- Posts: 3650
- Joined: 13 Jan 2014 23:30
- Location: Indianapolis, Indiana USA
- Contact:
Re: Working with large black/whitelists
Oh, if anyone uses this, you might want to make the following change
from:to:
I just realized this would be useful when I forgot the parameter to get the whole list to show. Oops.
from:
Code: Select all
$table_html = "Results hidden\n";
Code: Select all
$table_html = "Results hidden. Click <a href='./lists.php?showall'>here to show all</a>\n";