commit 142d151f075a73706f914451bceef18f4f07c28f
author Steven <sluong@cisco.com> Fri May 26 14:18:58 2017 -0700
committer Damjan Marion <dmarion.lists@gmail.com> Mon May 29 10:34:47 2017 +0000
tree d7be7b2be213b43fbde4ade03455747a70018185
parent 3474f9adca19dbd8ab41460f8e891924beaa0210

memif: master instance crashes when typing quit on slave

When I type in 'quit' on the slave instance, the master instance crashes
on this line.

0: /home/sluong/vpp-master/vpp/build-data/../src/vlib/unix/input.c:200 (linux_epoll_input) assertion `! pool_is_free (um->file_pool, _e)' fails
Aborted (core dumped)

Below is the decode from gdb
    line_number=0, fmt=0x7f57af6cc9a0 "%s:%d (%s) assertion `%s' fails")
    at /home/sluong/vpp-master/vpp/build-data/../src/vppinfra/error.c:143
    vm=0x7f57af8e2400 <vlib_global_main>, node=0x7f576d40ad80, frame=0x0)
    at /home/sluong/vpp-master/vpp/build-data/../src/vlib/unix/input.c:200
    vm=0x7f57af8e2400 <vlib_global_main>, node=0x7f576d40ad80,
    type=VLIB_NODE_TYPE_PRE_INPUT, dispatch_state=VLIB_NODE_STATE_POLLING,
    frame=0x0, last_time_stamp=1525665215050617)
    at /home/sluong/vpp-master/vpp/build-data/../src/vlib/main.c:1016
    vm=0x7f57af8e2400 <vlib_global_main>, is_main=1)
    at /home/sluong/vpp-master/vpp/build-data/../src/vlib/main.c:1500

I am able to reproduce the problem consistently with the below procedure.

1. Create 3 memif interfaces between slave and master instances.
2. Type 'quit' on the slave. Neither crashes the first time.
3. Bring back the slave. Type 'quit' on the master. Neither crashes.
4. Bring back the master. Type 'quit' on the slave. The master crashes.

There are two places the interrupt line is disconnected and the unix file is
removed via the call unix_file_del ()

1. memif_int_fd_read_ready ()
2. memif_disconnect () which is called via multiple places in memif.

When the crash happens, the unix file was removed from memif_disconnect ()
via memif_conn_fd_read_ready () with size of the message == 0 in recvmsg ().
It is noted when the unix file was removed from memif_int_fd_read_ready (),
it never crashes. It is a race condition. However, if I follow the
aformentioned procedure, the crash always happens.

The reason the crash happens when memif_disconnect () removes the unix file
is because there may still be pending input in linux_epoll_input (). When
linux_epoll_input () tries to access the unix file via the line 200
      unix_file_t *f = pool_elt_at_index (um->file_pool, i);
it crashes.

We could add code in linux_epoll_input () to avoid the crash if the index
for the particular file_pool is already free. Or we could fix memif to not
remove the unix file in memif_conn_fd_read_ready () when recvmsg () got 0
byte and just postpone the unix file deletion in memif_int_fd_read_ready ()
later after linux_epoll_input () got a chance to run to empty the input
stream.

I choose to fix the problem in the latter approach. I split the function
memif_disconnect () into two parts. For the code path which
memif_conn_fd_read_ready () calls memif_disconnect (), it does not remove the
unix file. All other calls to memif_disconnect () will continue to do what
it uses to do to avoid regression.

Please let me know if I should fix the problem other way.

Change-Id: I8efe2a3d24c6581609bc7b6fe82c2b59c22d8e4b
Signed-off-by: Steven <sluong@cisco.com>