Working with large black/whitelists

Request and discuss new features you would like to have.
Post Reply
maxkmv
Posts: 35
Joined: 28 Apr 2015 14:40

Working with large black/whitelists

Post by maxkmv » 21 Nov 2017 08:33

Greetings!
I work with many dozens of thousands of white/blacklist entries. Every time I want to add a new entry, it has to reload the entire page, this takes between 3 and 6 minutes (depending on my bandwidth). Is it possible to make a change so that it adds new entry to white/blacklist but doesn't have to reload the whole lot? For example, to display a message - "click here to load the entire list" or something similar? This will enable me to add good/bad IP addresses much quicker into my lists.

User avatar
shawniverson
Posts: 2442
Joined: 13 Jan 2014 23:30
Location: Rushville, Indiana, USA
Contact:

Re: Working with large black/whitelists

Post by shawniverson » 21 Nov 2017 15:15

Sounds like a good feature request :) I am noting this to engage MailWatch development.
Version 3.0.2.5 released! Update now to keep your eFa secure!

User avatar
pdwalker
Posts: 875
Joined: 18 Mar 2015 09:16

Re: Working with large black/whitelists

Post by pdwalker » 21 Nov 2017 15:17

Should be an easy change. Let me have a look and tell you what to do tomorrow.

budy
Posts: 42
Joined: 10 Sep 2017 07:33

Re: Working with large black/whitelists

Post by budy » 21 Nov 2017 15:44

Given the transient nature of the internet in general, do you really feel this is a very good idea to pursue?

User avatar
pdwalker
Posts: 875
Joined: 18 Mar 2015 09:16

Re: Working with large black/whitelists

Post by pdwalker » 21 Nov 2017 17:25

?

I'd find it useful. There are a few interface tweaks that I make to the mailwatch ui to speed up the interface. For example, when I run message filters to select particular messages, do I really need to know how many messages are going to appear before I go to the message list? Nah, no need to hold up the ui just to do a "select * from maillog where whatever" query.

User avatar
pdwalker
Posts: 875
Joined: 18 Mar 2015 09:16

Re: Working with large black/whitelists

Post by pdwalker » 21 Nov 2017 17:41

Easy temporary fix for you:

Code: Select all

[root@efa mailscanner]# diff -c lists.php.org lists.php
*** lists.php.org	Wed Nov 22 01:27:25 2017
--- lists.php	Wed Nov 22 01:28:18 2017
***************
*** 30,35 ****
--- 30,37 ----

  html_start(__('wblists07'), 0, false, false);

+ if (isset($_GET['showall'])) { $showall=true; }
+
  if (isset($_GET['type'])) {
      $url_type = deepSanitizeInput($_GET['type'], 'url');
      if (!validateInput($url_type, 'urltype')) {
***************
*** 302,309 ****
   * @param string $list
   * @return array
   */
! function build_table($sql, $list)
  {
      $sth = dbquery($sql);
      $table_html = '';
      $entries = $sth->num_rows;
--- 304,312 ----
   * @param string $list
   * @return array
   */
! function build_table($sql, $list, $showall=false)
  {
+ if ($showall) {
      $sth = dbquery($sql);
      $table_html = '';
      $entries = $sth->num_rows;
***************
*** 325,330 ****
--- 328,336 ----
      } else {
          $table_html = __('noentries07') . "\n";
      }
+ } else {
+ 	$table_html = "Results hidden\n";
+ }

      return array('html' => $table_html, 'entry_number' => $entries);
  }
***************
*** 412,422 ****

  $whitelist = build_table(
      'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
!     'w'
  );
  $blacklist = build_table(
      'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
!     'b'
  );
  echo '</table>
     </form>
--- 418,430 ----

  $whitelist = build_table(
      'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
!     'w',
!     $showall
  );
  $blacklist = build_table(
      'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
!     'b',
!     $showall
  );
  echo '</table>
     </form>
tl;dr; hate reading diffs

0/ edit lists.php

1/ add this code after line 32

Code: Select all

if (isset($_GET['showall'])) { $showall=true; }
2/ change this line:

Code: Select all

function build_table($sql, $list)
to this:

Code: Select all

function build_table($sql, $list, $showall=false)
3/ change this section from this:

Code: Select all

function build_table($sql, $list, $showall=false)
{
    $sth = dbquery($sql);
to this:

Code: Select all

function build_table($sql, $list, $showall=false)
{
if ($showall) {
    $sth = dbquery($sql);
4/ change this section from this:

Code: Select all

    } else {
        $table_html = __('noentries07') . "\n";
    }

    return array('html' => $table_html, 'entry_number' => $entries);
to this (yes, I was too lazy to change the indentation of the whole function):

Code: Select all

    } else {
        $table_html = __('noentries07') . "\n";
    }
} else {
        $table_html = "Results hidden\n";
}

    return array('html' => $table_html, 'entry_number' => $entries);
5/ and finally change this:

Code: Select all

$whitelist = build_table(
    'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
    'w'
);
$blacklist = build_table(
    'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
    'b'
to this:

Code: Select all

$whitelist = build_table(
    'SELECT id, from_address, to_address FROM whitelist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
    'w',
    $showall
);
$blacklist = build_table(
    'SELECT id, from_address, to_address FROM blacklist WHERE ' . $_SESSION['global_list'] . ' ORDER BY from_address',
    'b',
    $showall
To make your list reappear, just add a ?showall to your url, so from this:

Code: Select all

https://<efa-server>/mailscanner/lists.php
to this:

Code: Select all

https://<efa-server>/mailscanner/lists.php?showall
Note that is is just a temporary hack that will disappear on the next efa update. However, it's a trivial fix to add back. Also, if anyone thinks this should be a feature for mailwatch, this is not the right way to do it. It should be something configurable in the settings and properly localized.

maxkmv
Posts: 35
Joined: 28 Apr 2015 14:40

Re: Working with large black/whitelists

Post by maxkmv » 22 Nov 2017 09:35

Wow guys, thanks very much! I will try it out now!

maxkmv
Posts: 35
Joined: 28 Apr 2015 14:40

Re: Working with large black/whitelists

Post by maxkmv » 22 Nov 2017 10:16

Works great! :dance:

User avatar
shawniverson
Posts: 2442
Joined: 13 Jan 2014 23:30
Location: Rushville, Indiana, USA
Contact:

Re: Working with large black/whitelists

Post by shawniverson » 24 Nov 2017 16:35

Version 3.0.2.5 released! Update now to keep your eFa secure!

User avatar
pdwalker
Posts: 875
Joined: 18 Mar 2015 09:16

Re: Working with large black/whitelists

Post by pdwalker » 27 Nov 2017 04:50

Oh, if anyone uses this, you might want to make the following change
from:

Code: Select all

$table_html = "Results hidden\n";
to:

Code: Select all

$table_html = "Results hidden.  Click <a href='./lists.php?showall'>here to show all</a>\n";
I just realized this would be useful when I forgot the parameter to get the whole list to show. Oops.

Post Reply